HIVE-29781: Iceberg: Basic stats fetch optimization - bulk partition stats read with query-scoped cache - #6662
Open
deniskuzZ wants to merge 3 commits into
Open
HIVE-29781: Iceberg: Basic stats fetch optimization - bulk partition stats read with query-scoped cache#6662deniskuzZ wants to merge 3 commits into
deniskuzZ wants to merge 3 commits into
Conversation
deniskuzZ
marked this pull request as draft
July 29, 2026 20:27
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
deniskuzZ
changed the title
deniskuzZ
marked this pull request as ready for review
July 31, 2026 09:38
deniskuzZ
changed the title
deniskuzZ
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes Iceberg basic statistics fetching by introducing bulk, query-scoped partition stats reads (with caching) so that a single partition-stats read can serve the whole query instead of one read per partition. It also aligns ANALYZE/statistics flows and COUNT(*)-from-stats optimization with the new storage-handler backed stats model (including evolved/unpartitioned partitions).
Changes:
- Add bulk basic-stats retrieval paths (
HiveStorageHandler#getAggrBasicStatsFor,getRowCount(...)) and use them in planner/optimizer stats collection. - Rework
BasicStatsto accept explicitly provided stats without mutating HMS partition parameters; add tests for precedence. - Add query validation to reject partition-scoped ANALYZE for tables with non-native partition support; update Iceberg stats reading, caching, and golden outputs accordingly.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| ql/src/test/org/apache/hadoop/hive/ql/stats/TestBasicStats.java | Adds coverage for provided basic-stats precedence and immutability of partition parameters. |
| ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java | Switches basic stats collection to use bulk storage-handler stats and table-level fallback. |
| ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java | Adjusts ANALYZE basic-stats collection to use table-scoped handler computation and non-native partition support behavior. |
| ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java | Makes NOSCAN handler-based stats computation table-scoped and safer for null returns. |
| ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStats.java | Introduces provided-stats construction path and avoids storage-handler-driven parameter mutation. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java | Rejects partition-clause ANALYZE where non-native partition support makes it unsafe. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/PrunedPartitionList.java | Clarifies semantics of “referred partition columns” used by pruning logic. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java | Rejects partition-scoped column-stats ANALYZE for non-native partition support tables. |
| ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java | Adds shared validation helper for unsupported partition clauses. |
| ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java | Updates COUNT(*)-from-stats path to use storage-handler row count APIs for non-native tables. |
| ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java | Extends/adjusts storage handler API for table/partition row counts and batched basic stats retrieval. |
| ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java | Adds a canonical synthetic “unpartitioned” partition name constant for legacy rows. |
| ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java | Updates DESCRIBE to use the new table-scoped storage handler basic stats API. |
| iceberg/iceberg-handler/src/test/results/positive/truncate_partitioned_iceberg_table.q.out | Updates expected DESCRIBE output based on new stats population. |
| iceberg/iceberg-handler/src/test/results/positive/row_count.q.out | Updates expected EXPLAIN output reflecting revised row-count derivation. |
| iceberg/iceberg-handler/src/test/results/positive/iceberg_truncate_partition_with_evolution.q.out | Updates expected plans/outputs for evolved partitioning and revised stats behavior. |
| iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStatistics.java | Adds end-to-end tests for bulk stats, ANALYZE rejection, deletes, and evolved/unpartitioned stats behavior. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java | Refactors partition stats reading to a single-pass file read keyed by consistent partition names. |
| iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java | Implements bulk partition basic stats, query-scoped cache, and exact row-count APIs for rewrite decisions. |
| common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java | Adds a dedicated error message for rejecting partition-clause ANALYZE in the new validation path. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+965
to
+967
| if (tbl.isPartitioned()) { | ||
| PrunedPartitionList prunedList = pctx.getPrunedPartitions(tsOp.getConf().getAlias(), tsOp); | ||
| if (!prunedList.getReferredPartCols().isEmpty()) { |
Member
Author
There was a problem hiding this comment.
The zero was deliberately moved inside the referredPartCols check. Hoisting it reintroduces a wrong answer on Iceberg tables converted from unpartitioned to partitioned: partition enumeration skips the former unpartitioned spec, so getPartitions() is empty while the table still has rows.
With this change applied — SELECT count(*) returns 0 while SELECT * returns 3 rows.
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.
Suppressed comments (2)
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:252
- When the storage handler returns aggregated basic stats for only a subset of partitions, using
Map.of()as the default forcesBasicStatsto ignore any existing HMS partition parameters for the missing partitions (because an empty providedStats is treated as authoritative). Passingnullfor missing entries letsBasicStatsfall back topartish.getPartParameters()for those partitions while still using the batched stats for partitions that were returned.
return inputs.stream()
.map(pi -> factory.build(pi,
aggrBasicStats.getOrDefault(pi.getPartition().getName(), Map.of())))
.toList();
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:924
metastoreRowCountcan throwNumberFormatExceptionwhen basic stats are marked up-to-date but theROW_COUNTparameter is missing (e.g.,computeBasicStatistics(..., quickStats=true)intentionally omits row count but callers still mark basic stats accurate). This method should returnnullwhen the row count is absent/unparseable, not throw.
private static Long metastoreRowCount(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Map<String, String> parameters = hmsTable.getParameters();
return StatsSetupConst.areBasicStatsUptoDate(parameters) ?
NumberUtils.createLong(parameters.get(StatsSetupConst.ROW_COUNT)) : null;
}
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
One partition-statistics read per query instead of one per partition
Why are the changes needed?
Compile-time regression on partitioned Iceberg tables
Does this PR introduce any user-facing change?
No
How was this patch tested?
TestHiveIcebergStatistics.java